The present invention was created in response to the shortcomings of the current generation of image retouching systems. Other retouching systems use one of two methods for handling images: (1) high resolution/low resolution (high, res/low res), and (2) virtual image. Each of these two approaches overcomes some major obstacles, however neither fully responds to the needs of today's color professionals for high quality, and fast response at an affordable price.
In the high res/low res approach, the complete scanned image (referred to as the “high res” image) is subsampled to yield a much smaller image (referred to as the “low res” image). Because previous image retouching systems did not yield “real time” performance when handling large images (over 10M or 10 million bytes), it was necessary to invent an approach to allow the retouching system work on a smaller, i.e. low res image that would yield acceptable response times for the operator. Using this approach, retouching actions are stored in a script. When retouching is complete, the script is typically passed to a more powerful, and expensive, server and “executed.” That is, the actions contained in the script are applied to the high res image, which results in a high quality final image. The disadvantage of this approach is that the operator does not work with the actual image or at highly detailed levels (particularly for a magnified “close-up” of a portion). As a result, it is not always possible to perform highly detailed retouching actions such as silhouetting and masking. Moreover, unpleasant surprises may occur upon execution.
The virtual image approach, commonly used by desktop image editing packages (e.g. MacIntosh or Windows types), manipulates a copy of the actual image held in memory. In some cases, one or more copies or intermediate drafts are held, enabling the user to revert to a previous copy if an error is introduced. Using the virtual image approach, the image itself is transformed as retouching effects are applied.
The virtual image approach suffers two important shortcomings: first, large amounts of memory are required; and second, each effect is applied immediately to the entire image so that complex manipulation, such as large airbrushing, scaling and rotation, incur long processing delays.
Prior image retouching systems have used large mainframe computers or work stations and proprietary hardware. For example, U.S. Pat. No. 5,142,616, issued Aug. 25, 1992 to Kellas, et al., teaches an electronic graphic system. In this system, data relating to a user-defined low resolution image functions to control an image by the combining other image data with data defining a low resolution representation of the initial image. Once desired modifications have been achieved, the image is displayed on a display monitor so that a low resolution control image is converted to a high resolution representation. Stapleton, et al., U.S. Pat. No. 4,775,858, issued Oct. 4, 1988, also teaches the use of a large frame store to produce an image of higher resolution than that found on a television screen.
Due to the high amount of memory required for processing, personal computers have proven very slow and marginally acceptable. Moreover, even with larger mainframe systems, there is not always a good correlation between the monitor and the printed image since there is not always a way to visualize the final image on the display device. Thus, discrepancies can be introduced due to differences between screen resolution and print resolution. Other relevant patents include: U.S. Pat. No. 5,179,651 issued Jan. 12, 1993 to Taaffe, et al., U.S. Pat. No. 5,065,346, issued Nov. 12, 1991 to Kawai, et al., U.S. Pat. No. 4,656,467, issued Apr. 7, 1987 to Strolle, U.S. Pat. No. 4,833,625, issued May 23, 1989 to Fisher, et al., U.S. Pat. No. 4,288,821, issued Sep. 8, 1991 to Lavallee, et al., and U.S. Pat. No. 4,546,385, issued Oct. 8, 1985 to Anastassiou.
Numerous image processing procedures currently exist. Common to all procedures is modification of an image through recalculation operations to irreversibly rearrange dots or picture elements (“pixels”) of an original image (or those resulting from the most recent modification) into a new arrangement.
Perhaps the greatest disadvantage of known procedures stems from the image that is displayed on the monitor not being identical to the image that will eventually be printed, rendering the operator unable to see the work as it will actually appear in print. Anomalies and discrepancies can therefore occur in the printed image. Known procedures cannot resolve the fact that the image displayed on the operator's monitor screen is in most cases vastly less defined than the scanned image held in the computer's memory. (This is untrue only in the case of small, low resolution images.) Resolution (as measured in dots per inch) of modern display monitors is far less than the resolution of printed color images.
A second and perhaps equally important disadvantage of known image processing techniques is that the image editing effects are applied sequentially, i.e. step-by-step. This incurs a severe degradation in the quality of the original image if many image editing effects are applied to the same portion of an image.
Operations carried out on an image usually require a high degree of processing power. If processing power is unavailable, then the time required to carry out the operation becomes unacceptably long, thus reducing the scope and sophistication of possible operations to be carried out on the image. For example, airbrush strokes are currently extremely limited in size as a result of the extreme processing power needed to calculated image changes.
The irreversible nature of image processing using known procedures precludes the operator from easily implementing any second thoughts. Presently, the only way to correct an airbrush stroke which does not achieve a desired effect is to superimpose a new stroke (instead of merely erasing the unsuccessful stroke). Alternatively, computers equipped with large memory can save intermediate steps. However, this requires a huge amount of memory (e.g., a single 8½″×11″×300 dots per inch (dpi) figure requires over 33 million bytes).
The present invention overcomes these shortcomings and permits rapid and powerful editing functions even on less powerful desktop computers, by employing at least one, more preferably two and most preferably three new and independent processes: preprocessing, image editing, and raster image processing.
The subject invention advantageously uses what I call a Functional Interpolating Transfer System (FITS) to greatly speed editing of an image on standard microcomputers, thus eliminating the need for expensive workstations or special hardware. FITS breaks down image processing into three steps: preprocessing, image editing and FITS raster image processing. This results in a virtually instantaneous response and eliminates waiting for file saving or processing updates. With this technique, limits on file size and resolution disappear.
Preprocessing in the invention (brand name “FITS”) involves creating a specially formatted version of an image which allows image editing to progress at rapid speed.
Image editing refers to the process of retouching, combining or otherwise modifying images, to create the final desired image. Image editing involves, in the broadest sense, all processing operations performed on an original image. This includes the combining of images, effects such as sharpening, blurring, brightening, darkening, distortion, and include modifications to the color or appearance of all or part of a original image.
Color changes may be achieved in a variety of ways including global changes to the chromatic range of the image, or selective change to individual colors, e.g. changing blue to red.
Raster image processing (“RIP”) is performed in two instances: (1) each time a new screen view is generated for display on a monitor, and (2) when an output page is generated for the purpose of printing or incorporated into another system such as a desktop publishing system. FITS raster image processing combines the input images with the modifications generated in the second stage (image editing) to create either a screen or print image. The output image generated by the FITS RIP can have any resolution; thus it is said to be resolution independent.
FITS raster image processing (“FITS RIP”) involves taking the ensemble of image manipulations (the various steps or “layers” of changes) that are performed during the image editing process and computing a single image for purposes of printing or display on a monitor. Modifications to the image, made during image editing, are characterized in a manner that is independent of the resolution of the input images or final output image. During a FITS RIP, layers are first combined mathematically for each pixel or selected pixels in the desired image, rather than by applying each layer successively to the original images. For each final pixel, a single mathematical function is generated that describes the color, in an arbitrary color space, at that point. If, as preferred, only a sample of pixels are fully computed for each layer of change, the color values of intermediate pixels are computed by averaging the mathematical functions of the neighboring pixels and applying that function average to the original pixel's color, rather than simply averaging the color values of the surrounding pixels. This approach results in a time savings in overall image handling and a higher quality resulting image.
In the FITS approach, the image editing actions are characterized by parameters to mathematical functions and these are stored layer-by-layer in a file separate from the original image(s). Each intermediate modification to the image is effectively saved in a layer and each layer can be independently modified, deleted or reordered. The parameters can be stored for points in a grid that is itself independent of the “dots-per-inch” resolution (dpi) of either the original imported images, or the final output images. As a result, images for display or print can be generated at an arbitrary resolution.
Substantially less memory is required during image editing than with the virtual image approach since only the changes to each layer are stored, not entire image each time. As a result, a sophisticated, heavily retouched new image consisting of over 10 layers can be described in a FITS file of 2-5 megabytes (2-5M), as compared with over 30M (megabytes) for existing virtual image systems to store a 1 page new image (at 300 dpi resolution). Thus, the FITS approach yields a 10 to 1 average savings per page of image, and substantially more for larger images or higher resolutions. Note that 600 dpi images are now quite common for high quality publishing, and this is likely to increase in the future.
To sum up, current computerized image processing for obtaining a high definition image suffers from the dual disadvantages of requiring extremely high processing power, a limitation of productivity and creativity for the operator due to the irreversibility of image editing steps, and the quality restrictions inherent in a pixel-based approach.
The subject invention, on the other hand, provides a computerized image processing procedure which enables the operator to rapidly carry out advanced graphic operations, and to reverse decisions as required—without in any way affecting the definition or precision of the final image.